Update disaggregation metrics into input_tok_tpt/prefill gpu and output_tok_tpt/decode_gpu - #177
Conversation
📊 Line Count ReportFile: Total Lines: 956 Base Lines: 956 Change: No change ➡️ |
|
Lgtm |
There was a problem hiding this comment.
Pull Request Overview
This PR updates the benchmark metrics calculation to support disaggregated architectures by separating prefill and decode GPU counts. The changes enable accurate per-GPU throughput calculations for input tokens (prefill phase) and output tokens (decode phase) separately, rather than treating all GPUs uniformly.
Key Changes:
- Added separate GPU count tracking for prefill and decode operations from result filenames
- Updated throughput calculations to use appropriate GPU counts for each phase
- Modified result filename parsing to extract ctx (prefill) and gen (decode) GPU counts
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| utils/process_result.py | Added logic to read prefill/decode GPU counts from environment variables and calculate separate input/output throughput per GPU metrics |
| runners/launch_gb200-nv.sh | Updated git branch references and comment describing result filename format |
| .github/workflows/benchmark-multinode-tmpl.yml | Modified filename parsing to extract prefill_gpus and decode_gpus values and pass them to process_result.py |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| gpus=$(echo "$result_file" | sed -n "s/.*_gpus_\([0-9]*\).*\.json/\1/p") | ||
| prefill_gpus=$(echo "$result_file" | sed -n "s/.*_ctx_\([0-9]*\).*\.json/\1/p") | ||
| decode_gpus=$(echo "$result_file" | sed -n "s/.*_gen_\([0-9]*\).*\.json/\1/p") | ||
|
|
There was a problem hiding this comment.
Trailing whitespace on line 100. Remove the trailing spaces for cleaner code.
Replace the "Ported from NVIDIA/InferenceMAX PR #177" line with the detail that actually matters for reproducing the numbers: the runner, the srt-slurm pin, and the model-path resolution. 将"Ported from NVIDIA/InferenceMAX PR #177"一行替换为对复现结果真正有用的 信息：运行器、srt-slurm 固定版本以及模型路径解析方式。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix https://github.com/InferenceMAX/InferenceMAX/issues/131
Result files now contains prefill, decode and total number of gpus. Passing this info inside process_result.py to calculate separate IP and OP token TPT metrics.
Tests:
DISAGG:
https://github.com/InferenceMAX/InferenceMAX/actions/runs/19086559330
https://github.com/InferenceMAX/InferenceMAX/actions/runs/19084630695
AGG: https://github.com/InferenceMAX/InferenceMAX/actions/runs/19094453990
@functionstackx @cquil11